Test Range NetApp Build Document 


Purpose: This document describes the hardware, software, cabling and configuration for the 
NDB Test Range NetApp as built during NetApp SE onsite support visit. In addition, this 
document covers the configuration of the ESXi hosts and switches to support NetApp 
connectivity. 

Net App Hardware: 

• Model: FAS2552 

• Double Controllers 

• (24) 1.2T SAS disks 
Software: 

• NetApp Clustered Data ONTAP Release 8.3RC2 

• This software version makes more efficient use of storage space by striping root 
partitions across multiple disks. With this software version, each disk has two partitions: 
one small partition for root and one large partition for data. 

Physical Cabling: 

Followed the NetApp cabling guide for “switchless Clustered Data ONTAP” (guide included in the 
box with NetApp). All cabling is described in the table below: 


NetApp Port 

Connected Host/Interface 

Description 

Node 1 eOa 

EMPTY 


Node 1 eOb 

EMPTY 


Node 1 eOc 

SAN-SW-1 TE1 

IP Storage 

Node 1 eOd 

SAN-SW-2 TE1 

IP Storage 

Node 1 eOe 

Node 2 eOe 

Cluster Connection 

Node 1 eOf 

Node 2 eOf 

Cluster Connection 

Node 1 eOM 

TR-CORE-1 gl/5 

Wrench port (for mgmt cx) 

Node 1 SAS Oa 

Node 2 SAS Ob 

For Shelf-Shelf Comms 

Node 1 SAS Ob 

Node 2 SAS Oa 

For Shelf-Shelf Comms 

Node 2 eOa 

EMPTY 


Node 2 eOb 

EMPTY 


Node 2 eOc 

SAN-SW-1 TE2 

IP Storage 

Node 2 eOd 

SAN-SW-2 TE2 

IP Storage 

Node 2 eOe 

Node 1 eOe 

Cluster Connection 

Node 2 eOf 

Node 1 eOf 

Cluster Connection 

Node 2 eOM 

TR-CORE-1 gl/6 

Wrench port (for mgmt cx) 

Node 2 SAS Oa 

Node 1 SAS Ob 

For Shelf-Shelf Comms 

Node 2 SAS Ob 

Node 1 SAS Oa 

For Shelf-Shelf Comms 


Net App Configuration: 


• Node 1 and Node 2 configured as an Active/Passive Cluster named clusterOl. In an 


Active/Passive cluster, there is only 1 large data aggregate and it is assigned to Node 1. 
Node 2 will take over the data aggregate if Node 1 fails. Node 1 and Node 2 also each 
have their own root partition. 

NetApp can be managed through SSH or through web client ( https://10.9.8.30 ) 
Networking Configurations: 

• Create if_groups (2) - these create the Etherchannel for IP Storage connections 

• ala - eOc and eOd on Node 1 are members 

• a2a - eOc and eOd on Node 2 are members 

• both if_groups use Create Policy multimode LACP 

• Create Broadcast Domains (4) 

• Default 

• Physical interfaces eOa and eOb on both Nodes are assigned to 
this Broadcast Domain 

• This broadcast domain is just a parking lot for unused interfaces 

• IP space ‘Default’ assigned to this broadcast domain 

• Cluster 

• Physical interfaces eOe and eOf on both Nodes are assigned to 
this Broadcast Domain 

• This broadcast domain is used for Cluster communications 

• MTU is set to 9000 

• IP Space ‘Cluster’ assigned to this broadcast domain 

• ‘Cluster’ IP space is dynamically assigned 

• IP- Storage 

• if_group ala on Node 1 and a2a on Node 2 are assigned to this 
Broadcast Domain 

• MTU is set to 9000 for IP storage optimization 

• IP space ‘Default’ assigned to this broadcast domain 

• Management 


• Physical interfaces eOM on both Nodes are assigned to this 
Broadcast Domain 


• This broadcast domain is used for management traffic 

• IP space ‘Default’ assigned to this broadcast domain 

• Create Subnets (1) 

• IP-Storage - IP range 10.0.7.30 - 10.0.7.35 

• These IPs will automatically be allocated to a new Datastore when it is 
created 

• Create lifs - (4) Logical interfaces for Volumes 

• Iif_vmw_ds01 through Iif_vmw-ds04 created 

• lifs are needed so that each Volume is assigned its own IP 

• When a Volume is moved to a different node, the lif will move to the 
new Node, preventing suboptimal routing, for example Node l-> Node 
2-> Volume 

• Subnet IP-Storage assigned to lifs 

• Storage Virtual Machine vmw_svm assigned to lifs 

• Other Management Network Interfaces 

• Cluster Management - 10.9.8.30 

• Node 1 Management - 10.9.8.31 

• Node 2 Management - 10.9.8.32 

• Node 1 Service Port - 10.9.8.33 (this is an OOB console port) 

• Node 2 Service Port - 10.9.8.34 (this is an OOB console port) 

Create Aggregates - 2 root and 1 data 

• nl_aggr_root - Total space 368.42GB striped across 10 disks 

• Assigned to Node 1 

• n2_aggr_root - Total space 368.42GB striped across 10 disks 

• Assigned to Node 2 

• n3_aggr_data - Total space 19.59T striped across 23 disks 

• Assigned to Node 1 (Node 2 will take over if Node 1 fails) 

• All aggregates configured with RAID-DP 


• Enter License Keys 

• Cluster Base License 

• NFS License 

• CIFS License 

• iSCSI License 

• FCP License 

• Create a Storage Virtual Machine (SVM) - this is a virtual machine to manage the storage 

• vmw-svm - there are 3 main sections to configure: 

• Storage 

• Create Volumes (4 data and 1 root) 

• vmw_ds01 through vmw_ds04 created for data- 4TB 
each 

• Storage efficiency enabled (deduplication) 

• Security Style = UNIX 

• Vmwsvm_root is the root volume and storage efficiency 
is disabled 

• Mount Volumes to Namespace 

• Namespace is the root volume 

• Mount each Volume 

• Apply Default Export Policy to root volume 
vmwsvm_root 

• Apply vmw_policy Export Policy to data volumes 

• Create Export Policies (2) 

• Default - allows everyone read-only access to root volume via 
NFS protocol (Access Details - select all read-only) 

• vmw_policy - allows IP Storage subnet (10.0.7.0/24) full access 
to data volume via NFS protocol (Access Details - select UNIX 
read-write) 

• Configure Protocol Services - enable NFS for this SVM 


TR SAN-SWITCH Configuration 


• Set system MTU 900 to enable jumbo frames globally. Jumbo frames are recommended 
by NetApp to optimize performance on the storage network. 

• Port-channel 1 created to support Etherchannel for Node 1 IP Storage connectivity 

• Interfaces TE1/1/1 and TE2/1/1 

• channel-group 1 mode active 

• switchport access vlan 7 

• spanning-tree portfast 

• Port-channel 2 created to support Etherchannel for Node 2 IP Storage connectivity 

• Interfaces TE1/1/2 and TE2/1/2 

• channel-group 2 mode active 

• switchport access vlan 7 

• spanning-tree portfast 
TR-CORE-1 Configuration 

• Ports gl/5 and gl/6 configured in VLAN 98 for NetApp Management traffic 

• gl/5 - Node 1 management 

• gl/6 - Node 2 management 
TR ESXi Host Configuration 

• Vswitchl - 120 ports 

• MTU set to 9000 

• Active adapters - vmnic2, vmnic3 

• Load-balance - route based on originating virtual port 

• Vmkl - IP-storage networking interface 

• MTU set to 9000 

• VLAN ID 7 

• Load-balance - route based on originating virtual port 

• Active Adapters - vmnic2, vmnic3 

• Added (4) NetApp NFS datastores to the ESXi hosts in a cluster named cluster_ds 


• vmw_ds01 - 10.0.7.30 


vmw_ds02 - 10.0.7.31 


• vmw_ds03 - 10.0.7.32 

• vmw_ds04 - 10.0.7.33 

• Total storage capacity - 16TB 

Installed NetApp VSC plugin TR-monitor (10.9.8.22) and registered to vCenter server 

• Optimized NFS settings on ESXi hosts through VSC 

• Allows management of NetApp through VMware 
Installed NetApp VAAI plugin on each ESXi host via command line 

• From SSH prompt on ESXi server, type: 

• esxcli software sources vib list -d NetAppNasPlugin.zip 

• esxcli software vib install -n NetAppNasPlugin -d NetAppNasPlugin.zip 

• Enabled hardware acceleration on datastores through VMware 


